Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#2700 upgrade Prometheus to v2.31.1 + AlertManager to v0.23.0 #2763

Merged
merged 21 commits into from
Dec 7, 2021
Merged

#2700 upgrade Prometheus to v2.31.1 + AlertManager to v0.23.0 #2763

merged 21 commits into from
Dec 7, 2021

Conversation

seriva
Copy link
Collaborator

@seriva seriva commented Nov 30, 2021 •
edited
Loading

  • Updated Prometheus to 2.31.1
  • Updated AlertManager to 0.23.0
  • Added upgrade path for Prometheus and AlertManager

@przemyslavic
Copy link
Collaborator

/azp run

@przemyslavic
Copy link
Collaborator

✔️ Works fine for new deployments. Sending alerts from alertmanager via e-mail has been tested.
Tested on Ubuntu / RHEL / CentOS and CentOS on aarch64.

image

image

@seriva seriva marked this pull request as ready for review December 1, 2021 15:34
to-bar
to-bar reviewed Dec 2, 2021
View reviewed changes
Copy link
Contributor

@to-bar to-bar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will continue review tomorrow.

ansible/playbooks/roles/prometheus/tasks/upgrade/main.yml Outdated Show resolved Hide resolved
ansible/playbooks/roles/prometheus/tasks/upgrade/main.yml Outdated Show resolved Hide resolved
ansible/playbooks/roles/prometheus/tasks/upgrade/main.yml Outdated Show resolved Hide resolved
ansible/playbooks/roles/prometheus/tasks/upgrade/main.yml Outdated Show resolved Hide resolved
to-bar
to-bar previously approved these changes Dec 3, 2021
View reviewed changes
atsikham
atsikham previously approved these changes Dec 6, 2021
View reviewed changes
Copy link
Contributor

@atsikham atsikham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ansible/playbooks/roles/prometheus/defaults/main.yml Show resolved Hide resolved
@przemyslavic
Copy link
Collaborator

/azp run

@przemyslavic
Copy link
Collaborator

There is an issue upgrading Prometheus on RHEL/CentOS.
It fails when executing prometheus --version command:

2021-12-07T11:54:46.9921575Z [38;21m11:54:46 INFO cli.engine.ansible.AnsibleCommand - TASK [prometheus : Get Prometheus version] *************************************
2021-12-07T11:54:47.4005448Z [31;21m11:54:47 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-upgprmazceflannel-monitoring-vm-0]: FAILED! => {"changed": false, "cmd": "prometheus --version", "msg": "[Errno 2] No such file or directory", "rc": 2}

When the command is executed directly on monitoring vm there is no error:

[operations@ci-upgprmazceflannel-monitoring-vm-0 ~]$ prometheus --version
prometheus, version 2.10.0 (branch: HEAD, revision: d20e84d0fb64aff2f62a977adc8cfb656da4e286)
  build user:       root@a49185acd9b0
  build date:       20190525-12:28:13
  go version:       go1.12.5

@seriva seriva requested review from atsikham and to-bar December 7, 2021 14:48
@seriva
Copy link
Collaborator Author

seriva commented Dec 7, 2021

There is an issue upgrading Prometheus on RHEL/CentOS. It fails when executing prometheus --version command:

2021-12-07T11:54:46.9921575Z [38;21m11:54:46 INFO cli.engine.ansible.AnsibleCommand - TASK [prometheus : Get Prometheus version] *************************************
2021-12-07T11:54:47.4005448Z [31;21m11:54:47 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-upgprmazceflannel-monitoring-vm-0]: FAILED! => {"changed": false, "cmd": "prometheus --version", "msg": "[Errno 2] No such file or directory", "rc": 2}

When the command is executed directly on monitoring vm there is no error:

[operations@ci-upgprmazceflannel-monitoring-vm-0 ~]$ prometheus --version
prometheus, version 2.10.0 (branch: HEAD, revision: d20e84d0fb64aff2f62a977adc8cfb656da4e286)
  build user:       root@a49185acd9b0
  build date:       20190525-12:28:13
  go version:       go1.12.5

Pushed a fix.

@przemyslavic
Copy link
Collaborator

/azp run

@przemyslavic
Copy link
Collaborator

/azp run

@seriva seriva merged commit 9cee47c into hitachienergy:develop Dec 7, 2021
@plirglo plirglo changed the title #2700 upgrade Prometheus + AlertManager #2700 upgrade Prometheus to v2.31.1 + AlertManager to v0.23.0 Dec 9, 2021
@seriva seriva mentioned this pull request Dec 9, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@to-bar to-bar to-bar approved these changes

@przemyslavic przemyslavic przemyslavic approved these changes

@atsikham atsikham Awaiting requested review from atsikham

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@seriva @przemyslavic @atsikham @to-bar